Head mounted display device and method for providing visual aid using same

ABSTRACT

An external scene image captured by an external scene imaging electronic camera attached to a head mounted display (HMD) is projected and displayed onto an image display screen arranged in front of the eyes of the user as a virtual image with a suitable viewing distance corresponding to the visual acuity of the user. At this time, for each object image presented in the virtual image of the external scene image, the virtual image is processed and formatted to add a predetermined degree of binocular disparity and image blur to the virtual image projected and displayed on the right and the left image display screen on the basis of a predetermined converted distance calculated from the real distance of each object. Thus, the user is given a sense of a realistic perspective for the virtual image of the external scene, free of the discomfort or unease.

TECHNICAL FIELD

The present invention relates to a wearable-type information displaysystem, and more particularly to a head mounted display device(hereinafter, referred to as an HMD) that is mounted in a user's headportion and has a function for displaying predetermined videoinformation to the front side of user's eyes and an application systemthereof.

BACKGROUND ART

In recent years, information video display devices using so-called HMDsof a goggle type or a glass type mounted in a user's head portion havebeen spread rapidly.

Such an HMD has a function for projecting a virtual image having apredetermined visible distance seen from a user, and a necessaryinformation video can be presented to the user as a virtual image byusing this function.

Meanwhile, there are individual differences and variations in human'svisible ability. For example, generally, a near-sighted person hasvisual ability that is not different much from good visual ability in acase where an object located at a relative short distance such as a bookor a newspaper is viewed and has a characteristic in which the visualability rapidly decreases as a visual recognition object becomes farapart. To the contrary, a far-sighted person has a characteristic inwhich the visual ability is lower at a short distance than at a longdistance.

In addition, in a middle/old-age group of which population rapidlyincreases by reflecting an aging society of recent years, whilerelatively good visual ability is secured at an intermediate distance,visual ability for a short distance and a long distance rapidlydecreases. In other words, generally, a visible distance range of themiddle/old age group in which the visual ability is in some degree orhigher relative to a young age group tends to be narrowed. This isso-called presbyopia.

In consideration of differences and variations in the visual ability ofindividual persons, for example, in JP 2000-89157 A (Patent Document 1),an HMD configuration capable of changing a visible distance of a virtualimage such that a virtual image having an appropriate visible distancecan be projected according to a user's visual ability has beendisclosed.

CITATION LIST Patent Document

Patent Document 1: JP 2000-89157 A

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

When an HMD having a function for changing the visible distance of avirtual image as disclosed in Patent Document 1 is used, for example, byimaging an external scene viewed by a user using a video camera or thelike and performing projection display of a virtual image having avisible distance that is the most appropriate for the user's visualability, the external scene can be viewed by the user constantly in agood visual recognition state regardless of differences and variationsin the visual ability of each user. Thus, a visual aid function such asvisual correction glasses optimized for each user can be included in theHMD.

However, when videos of all the target objects within an imaging visualfield are simply projected to a same virtual image position, all thedepth perception senses for a target object that is sensed and acquiredwhen an external scene is viewed by the naked eyes are lost, and only amonotonous video can be visually recognized, whereby a very strangefeeling relative to that from a scene viewed by the naked eyes occurs.

An object of the present invention, in consideration of the problemsdescribed above, to provide a high-functionality visual aid systemcapable of securing a good visual recognition performance while depthperception similar to that of the case of naked eyes is maintained byusing an HMD.

Solutions to Problems

In order to solve the problems described above, the present invention,for example, employs configurations described in the claims. While thepresent application includes a plurality of means solving the problemsdescribed above, for example, there is provided a head mounted displaydevice that is mounted in a head portion of a user and displaysinformation videos in front of the eyes. The head mounted display deviceis configured to include: an external scene imaging unit that capturesan external scene in a visual field area including a directvisually-recognized visual field of the user; a virtual image projectorthat projects and displays the external scene video captured by theexternal scene imaging unit on left-eye and right-eye video displayscreens separately arranged in front of the eyes of the user as left-eyeand right-eye virtual image videos having a predetermined visibledistance; an within-visual-field target object distance detector thatdetects an actual distance up to a target object shown inside theexternal scene video captured by the external scene imaging unit fromthe user; and a perspective video generator that generates virtual imagevideos to which depth perception is added by performing a predeterminedtreatment of the left-eye and right-eye virtual image videos on thebasis of actual distance information of the target object detected bythe within-visual-field target object distance detector.

Effects of the Invention

According to the present invention, a visual aid system capable ofsecuring good visual ability in accordance with individual visualability while maintaining the depth perception by using an HMD can berealized.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view illustrating overviews of an HMD that is apremise of the present invention and an application system thereof.

FIG. 2 represents an explanatory diagram of virtual image projectionthat is a premise of the present invention.

FIG. 3 represents line drawings that illustrate relations between anactual distance up to a watching target object and relative sight at thetime of viewing using naked eyes that are a premise of the presentinvention.

FIG. 4 is a block configuration diagram of an HMD application visual aidsystem according to Embodiment 1.

FIG. 5 is an outline plan view for describing binocular parallax of avirtual image according to Embodiment 1.

FIG. 6 is an outline front view that illustrates an example of a virtualimage to which binocular parallax is applied according to Embodiment 1.

FIG. 7 is a line drawing that illustrates a relation among an actualdistance up to a watching target object, relative sight for a real imageacquired by seeing the target object through naked eyes, and relativesight of a virtual image according to Embodiment 1.

FIG. 8 is a line drawing that illustrates a relation between an actualdistance up to a watching target object and a scaled distance in avisual sense applied to a target object video inside a virtual imageaccording to Embodiment 1.

FIG. 9 is a flowchart that illustrates a process flow according toEmbodiment 1.

FIG. 10 is a block configuration diagram of an HMD application visualaid system according to Embodiment 2.

FIG. 11 is a block configuration diagram of an HMD application visualaid system according to Embodiment 3.

FIG. 12 is a block configuration diagram of an HMD application visualaid system according to Embodiment 4.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present invention will be described withreference to the drawings.

First, overviews of an HMD that is a premise of the present inventionand an application system thereof will be described. FIG. 1 is aperspective view illustrating overviews of an HMD and an applicationsystem thereof. Hereinafter, an HMD application system will be describedas a visual aid system compensating the visual recognition performanceof user's naked eyes.

In the case illustrated in FIG. 1 , the HMD 1 is mounted in a headportion of a user 20. The HMD 1 is a so-called video see-through typeHMD having a function for projecting and displaying a predeterminedvideo on a non-transparent video display screen (video screen device) 2arranged in front of the left and right eyes of a user 20. In otherwords, at least when the HMD 1 is in an operating state, video displayscreens 2 are in a non-transparent state, and accordingly, the user 20cannot visually recognize the outside through the naked eyes butvisually recognizes only videos that are projected onto the videodisplay screens 2.

In addition, in the HMD 1, an electronic camera 3 used for imaging anexternal scene arranged on the periphery of the naked eyes of the useris included in addition to the video display screens 2. The electroniccamera 3 used for imaging an external scene is installed to image anexternal scene in a direction that is almost the same as a visual linedirection of the user 20 wearing the HMD 1, has a function capable ofimaging an external scene in a visual field that is almost equal to orlarger than a naked-eye visual field of the user 20, in other words, afunction for imaging an external scene in a visual field area includinga direct visually-recognized visual field as electronic videoinformation, and has a sufficiently high imaging performance(resolution, MTF, and the like) relative to the visual recognitionperformance using the naked eyes of the user. In addition, theelectronic camera 3 may have a depth of field that is sufficiently deeprelative to a depth of field according to viewing through naked eyes.

Furthermore, in the electronic camera 3, an automatic focus functioncapable of automatically adjusting the focus to a predetermined targetobject within an imaging visual field is also included. As an object towhich the focus is adjusted may be a target object that is present atthe center of the imaging visual field, and, for example, as will bedescribed later, a certain visual line detecting device detecting thevisual line direction of a user may be arranged, and the focus may beadjusted to a target object present in the visual line direction. Inaddition, a device that images and stores a plurality of pieces of videoinformation having the focuses sequentially adjusted to a plurality oftarget objects having mutually-different distances from the user withinthe imaging visual field and generates a video called a “pan focus” or a“deep focus” having the focus adjusted over a wide distance range from ashort distance to a long distance by composing the plurality of piecesof video information may be included.

Furthermore, the electronic camera 3 may include a function for imagingan external scene by zooming up to a predetermined magnification ratioin accordance with an instruction from the user 20.

The HMD 1 has a distance measurement function for measuring a distanceup to each target object within the visual field that is imaged by theelectronic camera 3 used for imaging an external scene. As a specificexecution means of this distance measurement function, the automaticfocus function of the electronic camera 3 may be used, or a techniquefor detecting a distance from a parallax between videos of a targetobject that are independently captured by two electronic camerasarranged to be separate by a predetermined distance from each other maybe used. In addition, any means may be used such as additionalarrangement of a distance measurement sensor using infrared rays,ultrasonic waves, or the like as long as the means can realize apredetermined distance measurement function.

The videos of an external scene that are captured or generated asdescribed above are projected and displayed on the video display screens2 as a virtual image by using a virtual image projector included in theHMD 1.

At this time, for example, in a case where a target object 30 that is ata relative long distance such as a traffic sign illustrated in FIG. 2(a)is watched and, to the contrary, also in a case where a target object 40that is present at a relative short distance such as a newspaper or abook illustrated in FIG. 2(b) is watched, virtual images 31 and 41having an appropriate visible distance c at which a best visualrecognition performance is acquired by the user regardless of a distancebetween the user and the watching target object are projected anddisplayed. Accordingly, the user can visually recognize an externalscene similar to that at the time of viewing through naked eyesconstantly in a good visually recognized state regardless of differencesand variations in the visual ability of the user 20 in accordance withthe actual distances d and d′ of the watching target objects 30 or 40 orthe like.

Meanwhile, the appropriate visible distance c of a virtual imagedescribed above may be determined as below. FIG. 3 are examples in whichrelations between an actual distance up to a watching target object andrelative sight (a relative value of a target object visual recognitionperformance) at the time of visual recognition using naked eyes(hereinafter, such graphs will be described also as relative sightcharacteristic diagrams). In both the graphs, the horizontal axis takesan actual distance up to a watching target object, and the vertical axistakes a relative sight value of a case where sight of a case where aperson having good sight views a target object that is separate by adistance of 1 m through naked eyes is 1.0. Generally, in the case ofnaked-eye viewing with good sight, a relative sight characteristicdiagram, as denoted by a broken line 100 in each diagram, isapproximately in inverse proportion to the actual distance.

In contrast to this, a solid line 101 illustrated in FIG. 3(a)illustrates an example of a relative sight characteristic diagram of anear-sighted person. A near-sighted person has a small difference fromgood relative sight in a case where a distance up to a target object isrelatively short and has relative sight that tends to rapidly decreaseaccording to an increase in the distance. Thus, in a case where a userhas such a sight characteristic of near sight, for example, the actualdistance (in the example illustrated in FIG. 3(a), 0.8 m) of the targetobject located at a point P at which relative sight that is equivalentto relative sight of 1.0 in a good sight characteristic can be securedmay be set as the appropriate visible distance c of the virtual imagedescribed above.

On the other hand, a far-sighted person, although not illustrated in thedrawing, has relative sight that is equivalent to or better than goodrelative sight at a relatively long distance and has relative sight at ashort distance that tends to be markedly worsen than the good relativesight. Thus, also in a case where a user has such a sight characteristicof far sight, the actual distance of a target object (located at arelatively long distance) at which relative sight that is approximatelyequivalent to the good sight characteristic can be secured may be set asthe appropriate visible distance c of the virtual image.

A solid line illustrated in FIG. 3(b) illustrates an example of arelative sight characteristic diagram of a person having presbyopia. Inthe case of presbyopic sight, the sight tends to be worse than normalsight at a short distance and also at a long distance, and a point Q atwhich the relative sight is best is present near an intermediatedistance (in the example illustrated in FIG. 3(b), 0.8 m to 1.0 m).Thus, the actual distance of the target object of this Q point may beset as the appropriate visible distance c of the virtual image.

The technique for determining the appropriate visible distance c of thevirtual image described above is merely an example, and this embodimentis not limited to the determination method described above. In the humanrelative sight characteristic, there are large differenced andvariations due to various factors such as individual differences, agedifferences, an effect of the surrounding environment, and the like. Theappropriate visible distance c of a virtual image according to thisembodiment may be arbitrarily determined according to the visual abilityof each user.

In addition, by mounting a predetermined visual aid glass lenses betweenthe display screens 2 and the eyeballs of the user 20, the sight of theuser side may be configured to be corrected such that an arbitraryvisible distance becomes an appropriate visible distance.

In order to arbitrarily adjust the visible distance of a projectedvirtual image to an appropriate visible distance of each user asdescribed above, while a function for appropriately changing andadjusting the visible distance of the virtual image needs to beincluded. For such a function, for example, like the configuration asdescribed in Patent Document 1 described above, a known virtual imagevisible distance changing means may be used such as arrangement of adevice that mechanically moves at least a part of an optical systemgenerating the virtual image along an optical axis.

By using the configuration as described above, a user can utilize theHMD as a high-functionality visual aid system capable of visuallyrecognizing an external scene or a visual recognition target object wellregardless of a difference or a variation in the visual ability.

However, in a case where the HMD is utilized as the visual aid system byusing the technique described above, there is concern that an importantproblem described below occurs.

In other words, in a case where a captured external scene video isprojected and displayed as a virtual image in accordance with anappropriate visible distance of a user, the user can reliably visuallyrecognize an external scene constantly in a good visual recognitionstate regardless of the distance up to the watching target object.However, since all the target object videos within a visual field aredisplayed at the same visible distance as virtual images, a depthperception sense according to the actual distance of each target objectis completely lost, and only a planar scene can be recognized. For thisreason, a user feels remarkable strangeness and unpleasantness inaccordance with the lack of depth perception unlike in naked-eyeviewing.

Thus, a new virtual image display method or a new virtual image displaydevice is necessary which is capable of eliminating a strange feeling oran unpleasant feeling from a user by reliably reproducing depthperception felt by the user in a case where an external scene is viewedthrough the naked eyes as possibly while maintaining a goodvisually-recognized state by projecting a captured external scene videoas a virtual image in accordance with an appropriate visible distance.Hereinafter, specific embodiments thereof will be described.

Embodiment 1

FIG. 4 is a block configuration diagram of an HMD application visual aidsystem according to this embodiment. Hereinafter, the function of eachunit illustrated in this block configuration diagram will be described.

In the case illustrated in FIG. 4 , external scene video informationcaptured by an electronic camera 3 used for imaging an external scene istransmitted to a within-visual-field target object distance detector 10and a perspective video generator 11.

The within-visual-field target object distance detector 10 has afunction for detecting an actual distance (an actual distance from auser to a target object) up to each target object photographed insidethe captured external scene video. In the example illustrated in FIG. 4, while an example in which distance information is detected using acaptured video or a distance measurement function included in theelectronic camera 3 used for imaging is illustrated, as described above,it is apparent that a dedicated distance measurement sensor or adistance measuring device other than the electronic camera 3 may be usedfor the detection of the distance information of a target object. Insuch a case, naturally, the within-visual-field target object distancedetector 10 is configured to receive an information signal from thedistance measurement sensor or the distance measuring device. Inaddition, the video information captured by the electronic camera 3 isstored in a memory 12 in a form accompanying distance information ofeach target object detected by the within-visual-field target objectdistance detector 10 as is necessary.

A perspective video generator 11 is a core part of the visual aid systemaccording to this embodiment and has a function for generating virtualimage videos having depth perception that are independently projectedand displayed on video display screens 2 arranged in front of the leftand right eyes of an HMD user 20 by using external scene videoinformation captured by the electronic camera 3, distance information ofeach target object detected by the within-visual-field target objectdistance detector 10, predetermined video information extracted from thememory 12 as is necessary, and the like. The depth perception providingfunction of this perspective video generator 11 will be described later.

A display video generated by the perspective video generator 11 istransmitted to a video display controller 13. This video displaycontroller 13 has a function for controlling a virtual image projectingdevice included in an HMD 1 used for appropriately projecting a displayvideo transmitted from the perspective video generator 11 on projectionscreens 2 included inside the HMD 1 as a virtual image.

In addition, this video display controller 13 may have a function forcontrolling an optical system for generating a virtual image includedinside the virtual image projecting device such that the visualperception distance of a virtual image can be changed and adjusted to apredetermined distance.

The function and the operation of each of the electronic camera 3 usedfor imaging an external scene, the within-visual-field target objectdistance detector 10, the perspective video generator 11, the memory 12,and the video display controller 13 described above are appropriatelycontrolled by the controller 14.

As described above, the perspective video generator 11 among blocksillustrated in FIG. 4 is a core part of this embodiment, and apredetermined processing treatment is performed for virtual image videosthat are independently projected and displayed on the left and rightvideo display screens 2 therein, and thus, a virtual image video fromwhich a predetermined depth perception seen from the user 20 is felt isgenerated. Hereinafter, a specific embodiment for generating depthperspective will be described.

First, in a case where a person visually recognizes a virtual image, inorder to allow the person to feel depth perception, the following threeelements are essential.

-   (1) binocular parallax-   (2) shading of a virtual image that corresponds to a distance-   (3) alleviation of a strange feeling accompanying convergence of    both eyes

First, (1) the binocular parallax represents generating a difference inpositions of videos (virtual images), which are viewed by the left andright eyes, in the in-plane direction in accordance with a distance froman observer to a target object. FIG. 5 illustrates an example thereof.Here, an actual distance from the user 20 wearing an HMD 1 to a watchingactual target object 50 present right in front of the user is denoted byd, and a visible distance c of virtual images 51 and 52 of the targetobject 50 projected onto left and right projection screens 2 by the HMD1 is denoted by c. In addition, a gap between the both eyes of the useris denoted by W. At this time, in order to cause the user to feel depthperception similar to that of the actual target object 50 by viewing thevirtual images 51 and 52 of the actual target object 50, the virtualimages 51 and 52 of the actual target object 50 viewed by the left andright eyes need to be projected onto the positions of intersectionsbetween at least straight lines (broken lines in the drawing) joining atleast the actual target object 50 and both the left and right eyes ofthe user and a virtual image visual recognition face that is separate bythe visible distance c.

In other words, as illustrated in FIG. 6 , the virtual image 51 of theactual target object 50 in the left eye needs to be projected onto aposition shifted from the center of the virtual image to the right sideby Δx, and a virtual image 52 in the right eye needs to be projectedonto a position shifted from the center of the virtual image to the leftside by Δx.

The relative shift amount Δx of the left and right virtual images,generally, is represented in the following equation using the actualdistance d, the virtual image visible distance c, and a gap W betweenboth the eyes of the observer.Δx=(W/2)×(1−c/d)  (1)

In this way, as the relative shift amount Δx, by applying an in-planedirection shift (parallax) to virtual image videos projected into boththe left and right eyes that is in approximately inverse proportion to adistance from an observer (an HMD user in the case of this embodiment)to a watching target object, it helps the observer to feel depthperception.

Next, (2) the shading of a virtual image corresponding to a distancerepresents application of a predetermined “shading” to a virtual imagevideo for each target object video within a projected virtual image incorrespondence with an actual distance of a target object consideringthat the human visual ability is approximately in inverse proportion toa distance up to a target object as described above.

FIG. 7 illustrates an example thereof. FIG. 7 , similar to FIG. 3 ,represents a human relative sight characteristic, and the horizontalaxis and the vertical axis are similar to those illustrated in FIG. 3 .As described above, relative sight for a real image acquired by directlyviewing a target object using naked eyes, as denoted by a broken line100 in the drawing, decreases approximately in inverse proportion to theactual distance of the target object.

On the other hand, in a case where a scene imaged by a camera is seen asa virtual image, the visual recognition position of the virtual image isfixed regardless of a visual recognition position of the virtual imagebeing near/far from the target object, and accordingly, the relativesight for the virtual image video of the target object, as denoted by asolid line 200 in the drawing, has a fixed value regardless of theactual distance up to the target object. This is an important factorblocking a depth perception sense of the virtual image.

Thus, by applying a “shading” of an amount that is approximately inproportion to the actual distance information of a target object to eachtarget object video shown inside a video projected as a virtual image inadvance, even a virtual image located at the same visible distanceapproaches the relative sight characteristic curve 100 for a real imageaccording to naked-eye viewing, and it can help the observer to feeldepth perception on the basis of the relative sight characteristiceffect.

In addition, at this time, in order to optimize the visibility of atarget object watched by a user, the “shading” amount corresponding toan actual distance up to the target object may be appropriately adjustedsuch that relative sight for the actual distance up to the watchingtarget object is optimized.

Furthermore, regarding the specifying of a watching target object, atarget object located at the center within the imaging visual field thatapproximately matches between the naked eyes of the user may bespecified as a watching target object of the user, or a visual linedirection of a user may be detected, and a target object present in thevisual line direction may be specified as a watching target object.

Finally, (3) the alleviation of a strange feeling according to theconvergence of both the eyes is as below. The convergence of both theeyes represents motions (inner rotational motions) of the eyeballs ofboth the eyes rotating to the inner sides for adjusting the focus ofboth the eyes to a target object in a case where, particularly, thetarget object located at a short distance is directly visuallyrecognized. As the brain detects such inner rotational motions of theeyeballs, a person detects depth perception to some degree. Suchconvergence markedly occurs in a case where the target object is locatedat a very short distance within several tens of centimeters. On theother hand, in a case where a person views a virtual image, the focus ofthe eyes of the person matches the visually recognized distance of thevirtual image. For this reason, in a case where a virtual image having avisible distance of about several tens of centimeters or more isvisually recognized, marked convergence does not occur. However, in acase where the target object is present at a very short distance fromthe observer, there is a deviation between the depth perception of avirtual image that is arbitrarily given according to the shading of thevirtual image corresponding to the binocular parallax described aboveand the distance and the like and a distance sense detected from theconvergence, and the observer strongly feels strangeness andunpleasantness.

An object of this embodiment is to eliminate the strangeness and theunpleasantness felt by the observer (HMD user). Thus, in thisembodiment, a technique for applying a predetermined correction of theactual distance of the target object and a scaled distance in a visualsense that is given to the virtual image is proposed.

FIG. 8 illustrates one example thereof. This diagram is a graph in whicha relation (hereinafter, this relation will be referred to as a distancescaling characteristic) between the actual distance up to a watchingtarget object and a scaled distance in a visual sense that is given toeach target object video inside the virtual image in this embodiment isplotted.

Originally, it is ideal that the actual distance (horizontal axis) up toa target object and a scaled distance (vertical axis) in a visual sensethat is given to a target object video inside the virtual imageprecisely match each other. In other words, it is ideal that thedistance scaling characteristic described above has a linear relationhaving a slope (a coefficient of proportion) of 1.0 as in a broken line110 illustrated in FIG. 8 . However, as described above, when such acorrect distance relation is maintained, particularly in ashort-distance area, there is a large deviation between the depthperception that is arbitrarily given to a virtual image as describedabove and a distance detected from the convergence.

Thus, in this embodiment, a function of calculating a scaled distanceaccording to a distance scaling characteristic on a curve, for example,as represented by a solid line 210 in the drawing in which a straightline 110 representing the original correct distance relation is followedin an intermediate distance/long-distance area and, in a short-distancearea located nearer than a predetermined point R (in the exampleillustrated in the drawing, an actual distance of 1 m), a coefficient ofproportion of the amount of change in the distance in the visual sensethat is given to a virtual image video with respect to the amount ofchange in the actual distance up to the target object is graduallydecreased from 1.0 in accordance with a decrease in the actual distanceup to the target object is included in the within-visual-field targetobject distance detector 10 or the perspective video generator 11.

Then, on the basis of the scaled distance calculated as described above,the perspective video generator 11 calculates a binocular parallaxamount given to virtual image videos used for the left eye and the righteye respectively projected onto the left-eye and right-eye video displayscreens 2 and the amount of shading of an image according to thedistance and performs a processing treatment of the videos to apply theamounts. By performing such a treatment, a deviation between the depthperception that is arbitrarily given to virtual image videos and theconvergence is decreased, and the strangeness and the unpleasantnessfelt by the observer (HMD user) can be eliminated.

However, in a case where the coefficient of proportion of the amount ofchange in the distance in the visual sense given to virtual images withrespect to the amount of change in the actual distance in theshort-distance area is gradually decreased from 1.0 in this way,naturally, the fidelity of the depth perception of virtual images withrespect to the depth perception of the actual target object deterioratesin that area. However, particularly, in a case where a short-distancetarget object is watched, as in a case where a book, a newspaper, or thelike is read, mostly only the target object is strongly watched, andthere is a rare case in which there is a concern about the depthperception with a surrounding scene. Accordingly, it is more practicalfor the visual aid system to eliminate the strangeness and theunpleasantness felt by the observer (HMD user) than to secure thefidelity of the depth perception of the target object.

The distance scaling characteristic 210 illustrated in FIG. 8 is merelyan example, it is apparent that this embodiment is not limited thereto.According to the individual characteristic or taste of the user orsurrounding environments or situations, the location (actual distance)of the inflection point R of the distance scaling characteristic or theshape of the distance scaling characteristic curve on a short-distanceside nearer than R may be arbitrarily set. Any characteristic may beused as long as the characteristic is a distance scaling characteristicin which the scaled distance in the visual sense that is given tovirtual images is longer than the actual distance up to the targetobject by a predetermined amount or a predetermined ratio on theshort-distance side from a predetermined distance point.

As described above, by performing a predetermined treatment for videosprojected onto the projection screens 2 of the HMD 1 arranged in frontof the left and right eyes of the user so as to alleviate a strangefeeling accompanying the (1) binocular parallax, (2) the shading ofvirtual images according to the distance, and (3) alleviation of thestrange feeling accompanying the convergence of both eyes, the user canvisually recognize an external scene with a good visual recognitioncharacteristic without causing a strange feeling and an unpleasantfeeling for the depth perception of virtual images.

Next, the process flow of the visual aid system according to thisembodiment will be described with reference to a flowchart illustratedin FIG. 9 . In the case illustrated in FIG. 9 , first, in Step 1 (S1),predetermined video information is sequentially acquired from theelectronic camera 3 used for imaging an external scene. Next, in Step 2,the actual distance information of each target object within the imagingvisual field is acquired. Next, in Step 3, by using the actual distanceinformation of each target object acquired in Step 2, a scaled distancein the visual sense that is given to each target object video inside avirtual image is calculated as a convergence in accordance with thedistance scaling characteristic described above. Next, in Step 4, the“binocular parallax amount” given to virtual image videos of the leftand right eyes is calculated, for example, by using Equation (1)described above on the basis of the visual sense scaled distance ofvirtual images scaled in Step 3.

Similarly, in Step 5, a predetermined “shading amount” to be given tovirtual image video of each target object is calculated on the basis ofthe visual sense of the virtual images scaled distance scaled in Step 3.

Next, in Step 6, by processing the video information acquired in Step 1by using depth perception building parameters such as the “binocularparallax” and the “shading amount” calculated in Steps 4 and 5, virtualimage videos to be projected onto the left and right video displayscreen 2 disposed inside the HMD 1 are generated.

Finally, in Step 7, left and right virtual image videos generated inStep 6 are projected and displayed onto the left and right video displayscreens 2 disposed inside the HMD 1. Then, in subsequent Step 8, it isdetermined whether or not the series of the process flow is to becontinued. When a result of the determination is “Yes”, the process isreturned to Step 1 again. On the other hand, in a case where a result ofthe determination is “No”, the series of the process flow ends.

When the process flow described above ends, a state is formed in whichno video is shown on the video display screens 2 arranged in front ofthe eyes of the user 20. These video display screens 2 are basicallynon-transparent, and it is difficult to visually recognize an outerscene using the naked eyes in a state nothing is shown, and accordingly,it is very dangerous unless the HMD is taken off. Thus, for example,these video display screens 2 are configured using liquid crystal-typelight control glass or the like, and switching between a transparentstate and a non-transparent state can be performed by turning on/off ofthe voltage. In this way, when the function of the visual aid system isin an operated state, the video display screens function asnon-transparent video display screens, and, when the operation of thesystem is turned off, the video display screens 2 are automatically in atransparent or semi-transparent state, and an external scene can bevisually recognized using the naked eyes, and it is safe to constantlywear the HMD.

In the HMD application visual aid system according to this embodimentillustrated in FIG. 4 , each block configuration part may be formedintegrally with or separately from the HMD 1. In addition, theappropriate visible distance C of virtual images may be set to bechangeable by the user.

As above, this embodiment is a head mounted display device that ismounted in a head portion of a user and displays information videos infront of the eyes and is configured to include: an external sceneimaging unit that captures an external scene in a visual field areaincluding a direct visually-recognized visual field of the user; avirtual image projector that projects and displays the external scenevideo captured by the external scene imaging unit on left-eye andright-eye video display screens separately arranged in front of the eyesof the user as left-eye and right-eye virtual image videos having apredetermined visible distance; an within-visual-field target objectdistance detector that detects an actual distance up to a target objectshown inside the external scene video captured by the external sceneimaging unit from the user; and a perspective video generator thatgenerates virtual image videos to which depth perception is added byperforming a predetermined treatment of the left-eye and right-eyevirtual image videos on the basis of actual distance information of thetarget object detected by the within-visual-field target object distancedetector.

In addition, there is provided a video display method of a head mounteddisplay device that is mounted in a head portion of a user and displaysinformation videos in front of the eyes. The video display methodincludes: capturing an external scene video in a visual field area ofthe user; detecting an actual distance up to a target object showninside the external scene video from the user; projecting and displayingthe captured external scene video on left-eye and right-eye videodisplay screens separately arranged in front of the eyes of the user asleft-eye and right-eye virtual image videos having a predeterminedvisible distance; and adding depth perception to the left-eye andright-eye virtual image videos on the basis of actual distanceinformation of the detected target object.

In this way, a visual aid system capable of securing good visualrecognition performance in accordance with an individual visual abilitywhile maintaining depth perception using an HMD can be realized.

Embodiment 2

FIG. 10 illustrates a block configuration diagram of an HMD applicationvisual aid system according to this embodiment. In this block diagram,the same reference numeral is assigned to the same constituent elementas that of the block diagram illustrated in FIG. 4 , and descriptionthereof will not be presented.

In this embodiment, in addition to the constituent elements ofEmbodiment 1 illustrated in FIG. 4 , a visual line direction detector 15used for detecting the visual line direction of the user 20 and awatching target object detector 16 that detects and specifies a targetobject at which the user current watches from the visual line directionacquired from the visual line direction detector 15 and an externalscene video captured by the electronic camera 3 are included. Then, the“shading amount” assigned to virtual image videos is adjusted such thatthe watching target object detected and specified by the watching targetobject detector 16 can be visually recognized with best visibility, andvirtual image videos to be projected onto left and right video displayscreens 2 are generated by the perspective video generator 11.

By employing such a configuration, even in a case where a user watches atarget object deviates from the center of the visual field, virtualimages of the watching target object can be viewed with an appropriatevisual recognition characteristic.

Embodiment 3

FIG. 11 illustrates a block configuration diagram of an HMD applicationvisual aid system according to this embodiment. In this block diagram,the same reference numeral is assigned to the same constituent elementas those of the block diagrams illustrated in FIGS. 4 and 10 , anddescription thereof will not be presented.

In this embodiment, in addition to the constituent elements ofEmbodiment 1 illustrated in FIG. 4 , a sound collecting microphone 17used for collecting speech given off by a user 20 and a speechrecognizer 18 recognizing a content of the speech given by the user 20by analyzing the speech of the user 20 collected by the sound collectingmicrophone 17 are included.

In this embodiment, for example, when a certain keyword such as“enlarge” or “zoom up” given off by the user 20 is detected, the keywordis recognized by the speech recognizer 18, and a function for zooming upa video captured by an electronic camera 3 used for imaging an externalscene at a predetermined magnification ratio may be provided.

Alternatively, inside the system described above, a separate characterrecognizer or an automatic predetermined language translator may beprovided, and, a function for automatically translating an English word,an English sentence, a difficult Chinese character, or the like presentwith the imaging visual field and displaying a result of the translationon virtual image videos shown on the display screens 2 in an overlappingmanner when a keyword such as “translate” is detected and recognized maybe provided.

In this way, by arranging a device detecting and recognizing speech of auser, the user can operate various functions relating to the HMD byusing only speech, and accordingly, a complete hands-free operation canbe performed.

Embodiment 4

FIG. 12 illustrates a block configuration diagram of an HMD applicationvisual aid system according to this embodiment. In this block diagram,the same reference numeral is assigned to the same constituent elementas those of the block diagrams illustrated in FIGS. 4, 10, and 11 , anddescription thereof will not be presented.

In this embodiment, in addition to the constituent elements ofEmbodiment 1 illustrated in FIG. 4 , a small-size automatic sightmeasuring unit (autorefractometer) 60 having a function forautomatically measuring the sight of the user 20 wearing the visual aidsystem according to this embodiment and an optimal virtual image visibledistance calculator 61 calculating a virtual image visible distance atwhich best visibility can be secured for the user on the basis of thesight of the user 20 that is detected by the automatic sight measuringunit 60 are included. Then, optimal virtual image visible distanceinformation calculated by this optimal virtual image visible distancecalculator 61 is transmitted to the perspective video generator 11 andthe video display controller 13, and a virtual image generating opticalsystem disposed inside the HMD 1 is automatically adjusted through thevideo display controller 13 such that virtual image videos maintaininggood depth perception not giving a strange feeling or an unpleasantfeeling to a user having the optimal virtual image visible distance aregenerated and are projected and displayed on the video display screens.

By employing such a configuration, every time when the user 20 wearsthis visual aid system, the sight of the user is automatically measured,and a virtual image position can be automatically adjusted to have avirtual image visible distance at which optimal visual recognitionperformance is acquired for the user.

Embodiment 5

In this embodiment, an example in which the process of the perspectivevideo generator 11 of the HMD application visual aid system isconfigured as an external process to reduce the processing burden of theHMD application visual aid system will be described.

As described in Embodiment 1 with reference to FIG. 4 , the perspectivevideo generator 11 has a function of generating virtual image videoshaving depth perception that are independently projected and displayedon the video display screens 2 arranged in front of the left and righteyes of the HMD user 20 by using the external scene video informationcaptured by the electronic camera 3, the distance information of eachtarget object detected by the within-visual-field target object distanceinformation detector 10, predetermined video information extracted fromthe memory 12 as is necessary, and the like and, as described above,performs a process of applying a predetermined treatment for videosprojected on the projection screens 2 of the HMD 1 arranged in front ofthe left and right eyes of the user to achieve the (1) binocularparallax, (2) shading of virtual images according to the distance, and(3) alleviation of a strange feeling accompanying the convergence ofboth the eyes. For this reason, a case may be considered in which theprocess of the perspective video generator 11, for example, has a heavyprogram load or a heavy processing speed load, and the HMD applicationvisual aid system, for example, becomes a bottleneck in a case where theHMD application visual aid system is integrated with the HMD 1 for adecrease in the weight.

For this reason, the process of the perspective video generator 11 maybe configured as an external process, and, for example, the externalscene video information and the distance information of a target objectare transmitted to a cloud through a network, the process of theperspective video generator 11 is performed on the cloud, and processeddisplay videos are configured to be received, processed by the videodisplay controller 13 of the HMD application visual aid system, anddisplayed on the video display screens 2 of the HMD 1.

In this way, the program load and the processing load of the HMDapplication visual aid support can be reduced.

The embodiments described above have been described in detail for easydescription of the present invention, and thus, all the describedconfigurations do not necessarily need to be included. In addition, apart of the configuration of a certain embodiment may be replaced by theconfiguration of another embodiment, and the configuration of a certainembodiment may be added to the configuration of another embodiment.Furthermore, for a part of the configuration of each embodiment,addition, omission, or replacement of another configuration may beperformed. In addition, the configurations or the functions of theembodiments may be combined together.

REFERENCE SIGNS LIST

-   1 HMD-   2 Video display screen-   3 External scene imaging electronic camera-   10 Within-visual-field target object distance information detector-   11 Perspective video generator-   15 Visual line direction detector-   16 Watching target object detector-   17 Speech microphone-   20 User-   30, 40, and 50 Target object-   31, 41, 51, and 52 Virtual image-   60 Automatic sight measuring unit-   61 Optimal virtual image visible distance calculator

The invention claimed is:
 1. A head mounted display device that displaysinformation videos in front of eyes of the user, the head mounteddisplay device comprising: an external scene imaging unit that capturesan external scene in a visual field area including a directvisually-recognized visual field of the user; a display controller thatdisplays the external scene video captured by the external scene imagingunit on left-eye and right-eye video display screens separately arrangedin front of the eyes of the user as left-eye and right-eye virtual imagevideos having a predetermined visible distance; a within-visual-fieldtarget object distance detector that detects an actual distance up to atarget object shown inside the external scene video captured by theexternal scene imaging unit from the user; and a perspective videogenerator that generates virtual image videos to which depth perceptionis added by performing a predetermined treatment of the left-eye andright-eye virtual image videos, wherein the perspective video generatorhas a function of calculating a virtual image visible distance thatapproximately matches the actual distance in a case where the actualdistance from the user to the target object is longer than apredetermined set distance and has a value larger than the actualdistance in a case where the actual distance is shorter than the setdistance on the basis of actual distance information of the targetobject detected by the within-visual-field target object distancedetector and a predetermined sight information of the user.
 2. The headmounted display device according to claim 1, further comprising a sightmeasuring unit that measures the sight information of the user, whereinthe perspective video generator calculates an optimal virtual imagevisible distance based on the actual distance information of the targetobject detected by the within-visual-field target object distancedetector and the sight information of the user measured by the sightmeasuring unit.
 3. The head mounted display device according to claim 1,wherein the display controller controls to automatically adjust thevisible distance between the user and the virtual image according to thecalculated virtual image visible distance.
 4. The head mounted displaydevice according to claim 2, wherein the display controller controls toautomatically adjust the visible distance between the user and thevirtual image according to the calculated optimal virtual image visibledistance.